Hash-based proximity clustering for efficient load balancing in heterogeneous DHT networks

Distributed hash table (DHT) networks based on consistent hashing functions have an inherent load uneven distribution problem. The objective of DHT load balancing is to balance the workload of the network nodes in proportion to their capacity so as to eliminate traffic bottleneck. It is challenging because of the dynamism, proximity and heterogeneity natures of DHT networks and time-varying load characteristics. In this paper, we present a hash-based proximity clustering approach for load balancing in heterogeneous DHTs. In the approach, DHT nodes are classified as regular nodes and supernodes according to their computing and networking capacities. Regular nodes are grouped and associated with supernodes via consistent hashing of their physical proximity information on the Internet. The supernodes form a self-organized and churn-resilient auxiliary network for load balancing. The hierarchical structure facilitates the design and implementation of a locality-aware randomized (LAR) load balancing algorithm. The algorithm introduces a factor of randomness in the load balancing processes in a range of neighborhood so as to deal with both the proximity and dynamism. Simulation results show the superiority of the clustering approach with LAR, in comparison with a number of other DHT load balancing algorithms. The approach performs no worse than existing proximity-aware algorithms and exhibits strong resilience to the effect of churn. It also greatly reduces the overhead of resilient randomized load balancing due to the use of proximity information. © 2007 Elsevier Inc. All rights reserved.